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Method For The Extraction of 

Image Features Caused by 
Structure Light Using Image 
Reconstruction 

Background of Invention 

3 [0001] The present invention relates to a system for observing objects in three 
I dimensional space using structured light. More particularly, the invention relates to a 

J multi-camera, three dimensional sensing system providing non-contact gauge 

measurements of an object using known object templates and known epi-polar 
geometry to identify observable projections of structured light on the surface of an 
object; and, to reconstruct corrupted portions of images of the projected structured 
light. 

[0002] As shown in Figure 1 , measurement systems such as laser range finders, 

illuminate an object undergoing measurement using structured light. Reflections of 
structured light projected on the surface of the object are captured by two or more 
calibrated cameras, generating images of illuminated portions of the object's surface. 
In some applications, structured light is in the form a set of laser planes. Where laser 
planes intersect the surface of the object, a striping effect is achieved. By detecting 
these laser light stripes in images of the object's surface, point correspondences can 
be established and triangulation techniques employed to reconstruct a representation 
of the surface of the object. 



[0003] 



First, a single pixel on each stripe in an image from a first camera is selected, as 
seen in Figure 2. Given that the position of a first camera lens in space is known and 
the selected pixel is known, then it is known that the point corresponding to the 
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selected pixel lies on a known line drawn from the lens center out into space. This line 
appears as a line in an image from a second camera. This line is called the epi-polar 
line of the selected point in the first image. Since the position of the lens center of the 
second camera is also known, this epi-polar line can be calculated and drawn in the 
second image. The epi-polar line, when drawn in the second image, will intersect at 
least one, and most likely several, of the stripes of the second video image. It is 
known that one of the pixels where the epi-polar line and a stripe intersect represents 
the selected point in the first image. The actual coordinate location in space of the 
point corresponding to any of these intersection points is determined by simple 
triangulation. Since the position in space of each plane of light which created the 
stripes is also known, the single point of all the intersection points which correspond 
to the selected point in the first image is ascertained by determining the three- 
dimensional coordinate of each intersection point to determine if it lies on one of the 
known planes of light. The intersection point which lies closest to a known plane of 
light is taken as the selected point. 

[0004] The surface of many objects includes regions which are shiny or have otherwise 
poor reflectivity characteristics. Structured light projected onto these surfaces results 
in multiple reflections or clutter causing poor laser stripe identification in generated 
images. Furthermore, in some situations, entire image regions which are 
representative of portions of an object's surface may become corrupted due to noise 
or interference. 

[0005] Accordingly, there is a need for a method of reconstructing projected laser stripes 
in those portions of images which are heavily corrupted due to reflection, noise, or 
interference. 

Summary of Invention 

[0006] Briefly stated, the present invention sets forth a method for generating a template 
structure representative of the surface of an object, and for utilizing the template 
structure to synthesize projected laser stripe data points in corrupted image regions, 
thereby improving the accuracy of current laser range finding and measurement 
systems. 
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[0007] The foregoing and other objects, features, and advantages of the invention as well 
as presently preferred embodiments thereof will become more apparent from the 
reading of the following description in connection with the accompanying drawings. 

Brief Description of Drawings 

[0008] In the accompanying drawings which form part of the specification: 

[0009] Figure 1 is a simplified view of a prior art three dimensional sensing system with 
which the present invention can be used; 

[001 0] Figure 2 is a simplified view of recorded images superimposed on a pictorial 

representation of light planes of the prior art three dimensional sensing system shown 
in Figure 1 , which produced the images; 

[001 1] Figure 3 is a simplified view of a image of laser stripes projected on a surface of 
an object, including both coherent and corrupted regions; 

[001 2] Figure 4 is a simplified diagram of a tangential flow field for a portion of the 
image shown in Figure 3; 

[001 3] Figure 5 is a simplified diagram of a perpendicular flow field for a portion of the 
image shown in Figure 3; 

[001 4] Figure 6 is a simplified view of a single laser stripe from an uncorrupted image 
region; 

[001 5] Figure 7 is a simplified view of a single laser stripe from a corrupted image region; 

[0016] Figure 8 is a collection of three images of a single laser stripe projected on a 
surface of an object, as seen from three individual cameras; and 

[001 7] Figure 9 is a before and after representation of image pixels defining local 

maxima in a corrupted region, illustrating improved coherent structure following 
reconstruction. 

[001 8] Corresponding reference numerals indicate corresponding parts throughout the 
several figures of the drawings. 
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Detailed Description 

[001 9] The following detailed description illustrates the invention by way of example and 
not by way of limitation. The description clearly enables one skilled in the art to make 
and use the invention, describes several embodiments, adaptations, variations, 
alternatives, and uses of the invention, including what is presently believed to be the 
best mode of carrying out the invention. 

[0020] Using a three dimensional measurement system such as the type shown in Figure 
1 , a set of canonical images is generated from repeated measurements of objects of 
the same type, so that the laser stripes in each image are manually identifiable. This 
process results in a set of templates which are representative of the surface of similar 
objects. Subsequent scans of similar objects will produce similar laser striping 
patterns, with any deviations from the template structures due to variations in the 
scanned object surface morphology. 

[0021] The template structures represent prior knowledge of the surface of the objects, 
such that features in subsequent images, such as seen in Figure 3, which are 
inconsistent with the expected results, as represented by the templates, can be 
ignored or de-emphasized. 

[0022] 

Predetermined template structures can be used in various ways to increase the 
fidelity of the laser stripe localization process. By using the template structures as a 
guide, a two-dimensional locally matched filter may be generated for each point in an 
image 1 0 of projected laser stripes 1 2A-1 2H on the surface of an object 1 3, such as 
shown in Figure 3. Next, a flow field is established which defines an orientation for 
each point in an image. The flow field, which may be either a tangential flow field, as 
seen in Figure 4, or a perpendicular flow field as seen in Figure 5, is generated using 
the templates by way of an orientation diffusion process, such as, but not limited to, 
interpolation methods and relaxation methods. Each arrow shown in Figures 4 and 5 
represents the assigned flow for a given pixel comprising the image. The filtering is 
done in two passes. In the first pass, each pixel (i,j) in an image is given the value: 

v(i, j) = 2 (ww^W x gaussian(r)) 

R 

where R is a curve which emanates from pixel (i j) and is always tangential to the flow 
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field, r is a measure of arc length along curve R, and image(r) is the image intensity 
value for a point on curve R. The gaussian term localizes this one dimensional filter. 

[0023] In the second pass, each pixel (i j) is given the value: 

K*> j) = X ( v W x gaussian(p)) 
p 

[0024] where Pisa curve emanating from pixel (i,j) and is always perpendicular to the 
flow field, and p is a measure of arc length along curve P. The result of this two pass 
approach is a two-dimensional local matched filter responsive to the original image 
10. The matched filtering enhances much of the true signal while suppressing 
unwanted noise. Alternatively, a single pass approach could also be used by 
employing a two-dimensional filter. Or, separable one-dimensional filters which are 
not gaussian could be employed within the scope of the invention. 

[0025] Once the image 1 0 has been processed with the filters to obtain values for v(i j) 
and t(i j) for each pixel, non-maximal suppression techniques are utilized to identify 
the centers of each laser stripe 1 2A 1 2H within image 1 0. In one embodiment, each 
raster line in an image is scanned to identify points where t(i,j) is a local maximum 
with respect to the raster line. These points represent the center of detected laser 
stripe structures in the raster line. In this way, the laser stripe signal to noise ratio is 
increased, resulting in an increase in measurement accuracy. 

[0026] As seen in Figure 3, due to problems associated with reflections and laser 

specularity, a region A in an image may become corrupted. A corrupted region is 
defined as a region wherein the projected laser stripes are either indistinct or are 
unstructured, i.e. the laser stripe is broken into a series of discontinuous points. To 
determine if a local region of an image is corrupted by either reflections or laser 
specularity, each pixel (ij) in the image (which is representative of a center point in a 
laser stripe projection 1 2, as defined by t(ij) being a local maxima, or zero-crossing) 
is identified. If pixels (i j) representative of local maxima in a region are in a structured 
pattern, i.e. form a line or are "chained", as seen in Figure 6, the region is considered 
to be coherent and not corrupted. Alternatively, if there are no pixels (i,j) 
representative of local maxima in a region, or if the representative pixels (ij) do not 
form a structured pattern, i.e., are incoherent, as seen in Figure 7, the region is 
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marked as corrupted. 

[0027] As seen in Figures 3 and 6, using the generated template structure and known 
epi-polar line geometry, the portions of a laser stripe in non-corrupted regions of 
image 10 can be readily identified. Points on the laser stripe passing through a 
corrupted region "A" in image 1 0 cannot be accurately identified from a single image. 

[0028] In one embodiment, shown in Figure 8, two or more uncorrupted images 14, 16 of 
the corrupted laser stripe region A, obtained by two or more different cameras located 
at different positions and orientations, are utilized to synthesize data points 
representative of the laser stripe using triangulation and projection. As seen in Figure 
8, a point in the corrupted region A of a first image 1 0 can be synthesized if it is 
q visible in uncorrupted regions of two or more separate images 14, 16. First, a target 

!|f point in the corrupted region A of the first image is identified using the template 

III structure for the object. Next, using epi-polar geometry and the known template 

f% structure, a corresponding point is identified on a laser stripe 1 2 in an uncorrupted 

'H image 14. The corresponding point in the first uncorrupted image 14 is used, 

i together with epi-polar geometry, to find the associated laser line in the second 

|H uncorrupted image 16. 

ifj [0029] The position and orientation of the cameras generating the first and second 
W uncorrupted images 1 4, 1 6 is known, hence each point on the laser stripe 1 2 

observed by the two cameras is known to lie on two separate lines. The three- 
dimensional location of each of the points is known to be the intersection of the two 
lines and can be determined by triangulation. Using the known position of a point in 
an uncorrupted image which corresponds to a data point of a corrupted image, 
together with the known camera position for the camera which generated the 
corrupted image, the position of a data point representing the target point on the 
laser stripe in the corrupted image can be accurately synthesized through the process 
of projection. 

[0030] Figure 9 illustrates the location of pixels identified as local maxima in a corrupted 
region of an image (shown as X's) together with the locations of synthesized points 
(shown as a dashed line). As can be seen the pixels identified as local maxima are 
unstructured, or incoherent, forming a wavy line, while the points representative of 
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the laser stripe synthesized from uncorrupted images form a structured or coherent 
straight-line structure, representative of the laser stripe location in the image. 

[0031] In view of the above, it will be seen that the several objects of the invention are 
achieved and other advantageous results are obtained. As various changes could be 
made in the above constructions without departing from the scope of the invention, it 
is intended that all matter contained in the above description or shown in the 
accompanying drawings shall be interpreted as illustrative and not in a limiting sense. 
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